Evaluation of a Mobile-Based Immunization Decision Support System for Scheduling Age-Appropriate Vaccine Schedules for Children Younger Than 2 Years in Pakistan and Bangladesh: Lessons From a Multisite, Mixed Methods Study

Background Missed opportunities for vaccination (MOVs), that is, when children interact with the health system but fail to receive age-eligible vaccines, pose a crucial challenge for equitable and universal immunization coverage. Inaccurate interpretations of complex catch-up schedules by health workers contribute to MOVs. Objective We assessed the feasibility of a mobile-based immunization decision support system (iDSS) to automatically construct age-appropriate vaccination schedules for children and to prevent MOVs. Methods A sequential exploratory mixed methods study was conducted at 6 immunization centers in Pakistan and Bangladesh. An android-based iDSS that is packaged in the form of an application programming interface constructed age-appropriate immunization schedules for eligible children. The diagnostic accuracy of the iDSS was measured by comparing the schedules constructed by the iDSS with the gold standard of evaluation (World Health Organization–recommended Expanded Programme on Immunization schedule constructed by a vaccines expert). Preliminary estimates were collected on the number of MOVs among visiting children (caused by inaccurate vaccination scheduling by vaccinators) that could be reduced through iDSS by comparing the manual schedules constructed by vaccinators with the gold standard. Finally, the vaccinators’ understanding, perceived usability, and acceptability of the iDSS were determined through interviews with key informants. Results From July 5, 2019, to April 11, 2020, a total of 6241 immunization visits were recorded from 4613 eligible children. Data were collected for 17,961 immunization doses for all antigens. The iDSS correctly scheduled 99.8% (17,932/17,961) of all age-appropriate immunization doses compared with the gold standard. In comparison, vaccinators correctly scheduled 96.8% (17,378/17,961) of all immunization doses. A total of 3.2% (583/17,961) of all due doses (across antigens) were missed in age-eligible children by the vaccinators across both countries. Vaccinators reported positively on the usefulness of iDSS, as well as the understanding and benefits of the technology. Conclusions This study demonstrated the feasibility of a mobile-based iDSS to accurately construct age-appropriate vaccination schedules for children aged 0 to 23 months across multicountry and low- and middle-income country settings, and underscores its potential to increase immunization coverage and timeliness by eliminating MOVs.


Introduction Background
Routine childhood immunization is the cornerstone of an efficient public health system and childhood disease prevention. Despite progress in improving routine immunization coverage, in 2022, 17% and 7% of children aged 0 to 23 months in Pakistan and Bangladesh, respectively, did not receive the third dose of the diphtheria-tetanus-pertussis vaccine [1]. The problem is further exacerbated for children who do visit health facilities in low-and middle-income countries (LMICs), but the immunization system fails to provide all age-appropriate vaccines to one in every 3 of these children [2]. This creates missed opportunities for vaccination (MOVs), where despite being eligible for vaccination, a child is not administered one or more of the vaccine doses [3]. MOVs remain a rampant problem in many LMICs [4,5], leading to underimmunization and delayed vaccination of children.
A systematic review of MOVs across LMICs showed a pooled MOV prevalence of 32.2% (95% CI 26.8%-37.7%) among children and 46.9% (95% CI 29.7%-64.0%) among women of childbearing age [2]. MOV prevalence estimates are even higher for some countries, ranging from 42% to 89% [6,7]. A study from Pakistan reported that despite being eligible for the pentavalent vaccine (Penta) at the measles vaccine visit, 34% of children did not receive the required dose of Penta even though they visited the immunization clinic and were vaccinated for measles [8]. Lack of awareness among parents and providers is one of the primary causes of MOVs, with vaccinator confusion about contraindications and the immunization schedule being the main factors [2,9]. In the absence of frequent refresher training, improper health care worker practices lead to repeated MOVs in LMICs and high-income countries [10,11]. However, regardless of the amount of training, scheduling age-appropriate vaccinations, especially catch-up schedules (when the child has missed or delayed one or more vaccines), is a complex task [12,13]. A study from Illinois showed that 33% of all health workers constructed incorrect hypothetical catch-up schedules in a survey designed to determine health workers' knowledge of catch-up immunizations [10]. In LMICs, where vaccinators are more likely to be overburdened and pressed for time [14], the likelihood of making errors in scheduling is even higher.
Decision support systems have proven to be powerful tools for improving clinical care and patient outcomes across various domains [15,16]. They have contributed to improving evidence-based medical practices, reducing medical errors, and improving adherence to standard care practices [17,18]. At the basic level, an immunization decision support system (iDSS) provides patient-specific vaccination recommendations at each immunization visit, considering the child's date of birth and vaccination history and ensuring the appropriate scheduling of feasible doses.
Currently, examples of iDSS deployment are limited and almost exclusively constitute complex systems from high-income country settings. For example, in the United States, clinical decision support systems based on national standard guidelines for immunization are embedded in regional immunization information systems [19] or web-based immunization forecasting services accessible by various electronic health records [20]. There is a dearth of literature on contextually appropriate, efficient, and practical mobile-based iDSS that is easily implementable within LMICs.

Objective
We aimed to analyze the diagnostic accuracy of an innovative mobile-based iDSS in an LMIC setting to construct age-appropriate vaccination schedules for children aged 0 to 23 months and compared with the gold standard of evaluation (World Health Organization [WHO]-recommended Expanded Programme on Immunization [EPI] schedule constructed by a vaccines expert). We also aimed to generate preliminary evidence for MOVs among visiting children (caused by inaccurate vaccination scheduling by vaccinators) that could be reduced through iDSS by comparing the manual schedules constructed by vaccinators with the gold standard. Finally, we reported the findings on the vaccinators' understanding, perceived usability, and acceptability of the iDSS.

Study Design
We implemented a sequential exploratory mixed methods study design with a quantitative component preceding the qualitative interviews. In the quantitative component, the children's demographic details and immunization history were recorded by trained study staff using iDSS. The iDSS used this information to formulate an age-appropriate immunization schedule for the children, and it was recorded on the back end and was not visible to the study staff (or vaccinators). Simultaneously, the study staff also captured the manually constructed immunization schedules determined by the vaccinators as indicated on the child's government-issued immunization cards. Through this process, we were able to capture both the iDSS and vaccinator schedules simultaneously for the same child (antigen doses). We used this information to assess the diagnostic accuracy of the iDSS algorithm by comparing the age-appropriate immunization schedules constructed by the iDSS for children aged 0 to 23 months with the gold standard of evaluation (WHO-recommended EPI schedule constructed by a vaccine expert). We also independently compared the vaccine schedules constructed manually by the vaccinators for the same children with the gold standard. This allowed us to generate preliminary evidence of MOVs resulting from inaccurate vaccination schedules constructed by vaccinators. This phase was followed by a qualitative phase in which the vaccinators were provided with iDSS-enabled study phones. After vaccinators had a chance to use the iDSS, we conducted in-depth interviews with vaccinators at the participating immunization centers regarding their experience of using the iDSS, its perceived utility, and acceptability.

Study Sites
We conducted a multicountry study at 6 immunization centers, 3 each in Pakistan and Bangladesh. Both countries vary in terms of full immunization coverage rate (Pakistan 66%; Bangladesh 84%) and infant mortality rates (81/1000 live births in Pakistan; 43/1000 live births in Bangladesh) [21,22]. Immunization centers were selected based on the influx of children, availability of vaccines and vaccinators, and absence of any kind of decision support system at these sites. The selected study sites had high penetration of mobile phones (>90%) and the presence of cellular networks (data connectivity) among the population [23,24].
Two of the selected immunization centers in Pakistan were located in Gilgit district in Gilgit Baltistan territory, which had a full immunization coverage rate of 57% among children aged 12 to 23 months in 2018 [21]. The third selected immunization center in Pakistan was a private center located in the Rahim Yar Khan district in Punjab province, with a full immunization coverage rate of 65% in 2019 [23].
In Bangladesh, all 3 immunization centers were located in Dhaka city in Dhaka district. The district had a full immunization coverage rate of 85% among children aged 12 to 23 months as of 2019 [25].

Study Population
The inclusion criteria for the study were as follows: children must be aged <2 years, visiting any of the 6 selected immunization centers for routine vaccination, and presenting with an immunization card. Exclusion criteria were as follows: children visiting for immunization campaigns and those who did not receive vaccination during their visit. We obtained verbal consent from the caregivers of the eligible children before enrollment.
For the qualitative component, our study population included all vaccinators from the participating immunization centers who used iDSS app during the study. We obtained written consent from the study participants before the interviews.

Ethics Approval
Ethics approval for the study was obtained from the Institutional Review Boards of the Interactive Research and Development (Pakistan; IRD_IRB_2019_01_010) and Building Resources Across Communities James P. Grant School of Public Health (Bangladesh).
For Bangladesh, the EPI schedule included BCG vaccine at birth; Penta, PCV, and OPV at 6, 10, and 14 weeks; IPV at 6 and 14 weeks; and measles-rubella (MR) vaccine at 9 and 15 months.

Development of the iDSS
We developed an android-based iDSS designed for mobile-based deployment, packaged in the form of an application programming interface (API) to function both as a stand-alone module and interoperable with other digital applications or platforms, such as web-based or mobile-based electronic immunization registries (EIRs).
The iDSS was formulated as a 2-step process comprising a data entry form and display interface showing the proposed vaccination schedule. In the first step, the details of the child, including the date of birth and immunization history, are entered in the data entry form of the iDSS. The algorithm uses this information to construct an age-appropriate immunization schedule (including the vaccines due at the current visit and those to be scheduled) tailored to the child and the respective country's EPI schedule. iDSS automatically reformulates the child's immunization schedule after every visit, adjusting for missed appointments and delayed vaccinations and considering the birth date, previous vaccines, standard interval based on live or inactivated vaccines and interdosing schedule, without the need for any manual calculation by the vaccinators. Changes in the EPI schedule and new vaccines are also incorporated into the iDSS algorithm. The results (current and scheduled vaccines) are instantly displayed on the iDSS interface through a color-coded system that allows for straightforward interpretation, especially by low literacy and overburdened vaccinators ( Figure 1). The iDSS proposes the dates for scheduled vaccines (accounting for public holidays and vaccine-specific days at centers) and prompts the vaccinator through warning messages if vaccines are being administered out of schedule to ensure that interdose gap guidelines are followed.
In addition, the iDSS algorithm can be customized to the needs of the country, according to the respective EPI schedule. The iDSS is also multilingual and is developed as open-source software for easy integration and interoperability across a variety of settings and systems. As part of the iDSS development, the module was pretested in-house for quality assurance and with randomly selected vaccinators (outside the study site) for limited field deployment. A detailed overview of the iDSS is provided in Multimedia Appendix 1.

Study Procedures and Data Collection
We enrolled eligible children whose accompanying caregivers provided verbal consent for participation. At enrollment, a unique study identifier was allocated to each child, and each follow-up visit by the child to the participating immunization center was recorded as a unique visit. Trained study staff used the iDSS installed on study mobile phones to record the demographics and immunization history of the enrolled children to enable iDSS to schedule their current and future vaccination visits. This information was recorded at the back end of the iDSS, and both the study staff and vaccinators were blinded to it. Simultaneously, the study staff also recorded the vaccination schedules determined by the vaccinators as per routine, which were obtained from the child's immunization cards. Vaccinators did not use the iDSS during the quantitative phase of the study. The iDSS app was linked to a web-based dashboard that allowed the real-time downloading of data for further analysis.
For the qualitative component of the study, we interviewed 16 vaccinators (11 from Pakistan and 5 from Bangladesh) after obtaining written consent. These vaccinators were provided with mobile phones with iDSS that they could use to schedule current and future vaccinations. Before using the iDSS, vaccinators were given a 2-day training by the study staff on using the iDSS module as part of their daily immunization activities. At the end of the period when the vaccinators had used the iDSS for at least 4 weeks, the study team interviewed the vaccinators. In-person interviews were conducted by Project Managers at both sites, who were qualified public health researchers with a medical background. The interviews were facilitated by Research Associates, who helped to take notes. We used a 16-item semistructured interview guide covering user experience, acceptability, and feasibility of the iDSS feature, and suggestions for improvement. Demographic indicators, including age, education, gender, and years of experience, were also collected from the participants. Each interview lasted between 25 and 40 minutes and was audio-recorded and transcribed for further analysis.

Study Outcomes
Our primary study outcome was the diagnostic accuracy of the iDSS in constructing age-appropriate vaccination schedules as per the WHO-recommended EPI guidelines. In addition, we examined the vaccinators' perceived usability and acceptability of the iDSS. A secondary outcome included preliminary estimates of the number of MOVs among children aged 0 to 23 months visiting study immunization centers for routine immunization visits.

Data Security
The phones used for data collection by the field staff and vaccinators had password locks with an additional level of protection through software sign-in passwords. Data collected through iDSS were uploaded to a secure database server. All the data sent from the phones to the server and back were encrypted in transit using the PBKDF2 algorithm (an industry standard). Access to data on the server was through a password-protected web dashboard interface, and only those involved with the project had access to the data for research purposes.

Quantitative Analysis
For descriptive analysis, we used frequencies and percentages for categorical data and means and SDs for continuous data. We compared the age-appropriate immunization doses scheduled by the iDSS and vaccinator with the gold standard. The latter helped determine the evidence for MOVs caused by inaccurate vaccination schedules constructed by vaccinators. To determine the gold standard vaccination schedule, an expert epidemiologist and practicing pediatrician reviewed the immunization history and date of birth or age of the child to determine the age-appropriate immunization doses due at each visit as per the WHO-recommended EPI schedule for the respective country. Summary statistics for the diagnostic tests were calculated using the diagt command package. Forest plots for sensitivity and specificity were generated using the metan command. The accuracy of the iDSS was determined along with the area under the receiver operating characteristic curves and their range. Analyses were performed using STATA software (version 17.0; StataCorp LLC).

Qualitative Analysis
The recordings of the qualitative data collected through the in-depth interviews were first transcribed and then translated into English. Transcriptions were coded by 2 researchers separately, who were public health practitioners, trained and experienced in performing qualitative analysis. The researchers extensively discussed and scrutinized the results to ensure the trustworthiness and comprehensiveness of the analysis. The final coding was shared with a third researcher to resolve any inconsistencies in the codes. Using a thematic analysis approach, the codes were sorted into categories to converge toward key overarching themes. Data were analyzed using NVivo qualitative data analysis software (version 12, 2018; QSR International).

Quantitative Analysis
From July 5, 2019, to April 11, 2020, a total of 6241 immunization visits were recorded from 4613 eligible children. A total of 73% (4557/6241) of visits were recorded from 3 immunization centers in Pakistan, and 27% (1684/6241) of visits were recorded from 3 immunization centers in Bangladesh ( Figure 2).
The proportion of male children (2435/4613, 52.8%) enrolled in the study was slightly higher than that of female children (2178/4613, 47.2%) across both the sites (Pakistan: 1720/3197, 53.8%) and Bangladesh (715/1416, 50.5%; Table 1). Most (3348/4613, 72.6%) children enrolled in the study were aged ≤6 months at the time of enrollment, both in Pakistan (2423/3197, 75.8%) and Bangladesh (925/1416, 65.3%). Table 2 shows the age-appropriate immunization doses scheduled by both the iDSS and vaccinators compared with the gold standard. In Pakistan, we collected data on 13,039 immunization doses for all antigens. The iDSS correctly scheduled 99.8% (13,015/13,039) of all age-appropriate immunization doses that should have been administered at the current visit compared with the gold standard, ranging from 99.2% to 100% for BCG vaccine, OPV-0 to 3, Penta-1 to 3, PCV-1 to 3, Rota-1 to 2, measles vaccine 1, and IPV vaccine doses. Of all the antigens, the proportion of correctly scheduled doses by iDSS was the lowest at 96.7% (202/209) for the measles-2 vaccine. In comparison, the vaccinator correctly scheduled 96.2% (12,545/13,039) of all the immunization doses that should have been administered on the current visit compared with the gold standard, ranging from 97% to 100% for BCG vaccine, OPV-0 to 3, Penta-1 to 3, PCV-1 to 3, Rota-1 to 2, and measles-2. However, the proportion of correctly scheduled doses by vaccinators due on the current visit decreased to 94.5% (362/383) and 66.3% (653/985) for measles-1 and IPV vaccine doses, respectively. Owing to errors in the vaccination schedules constructed by vaccinators, 3.8% (494/13,039) of all due immunization doses were missed by the vaccinators. Among the age-eligible children, the highest proportion of MOVs was for the polio and measles vaccines, where 33.7% (332/985) and 5.5% (21/383) of all due immunization doses, respectively, were missed by vaccinators.
In Bangladesh, we recorded data on 4922 immunization doses. The iDSS correctly scheduled 99.9% (4917/4922) of the age-appropriate immunization doses that should have been administered on the current visit compared with the gold standard, with the lowest proportion of correctly scheduled doses for the measles-2 vaccine (196/201, 97.5%). In comparison, the vaccinator correctly scheduled 98.2% (4833/4922) of all immunization doses that should have been administered on the current visit, ranging from 97.5% to 100% for BCG vaccine, OPV-1 to 3, Penta-1 to 3, PCV-1 to 3, IPV-1, and measles-rubella-2, but dropped to 95 (Figures 3 and  4).
Across both the sites, receiver operating curve analysis showed that the iDSS had high overall accuracy of scheduling age-appropriate immunization doses (area under the curve ranging from 99%-100% across antigens), whereas for vaccinator schedules, estimates for area under the curve ranged between 88% and 100%.
On the basis of the findings of the diagnostic accuracy of the iDSS in which the sensitivity and specificity were <100% for selected antigens, we updated the iDSS algorithm by fixing the errors related to the vaccine intervals and recommended age for administering the vaccines based on the WHO guidelines.    13039 Total a WHO-recommended EPI schedule constructed by a vaccine expert, using the following criteria: BCG at ≤1-year age; OPV-0 at ≤28 days; Penta-1, OPV-1, PCV-1, Rota-1, and IPV-1 at ≥6 weeks; Penta-2, OPV-2, PCV-2, and Rota-2 at ≥10 weeks and >28 days after vaccination with Penta-1, PCV-1, OPV-1, and Rota-1; Penta-3, OPV-3, and PCV-3 at ≥4 weeks age and >28 days after vaccination with Penta-2, PCV-2, and OPV-2; IPV-2 at ≥14 weeks age (for Bangladesh, >28 days after IPV-1); measles-1 at 9 months; measles-2 at 15 months and >28 days after measles-1 vaccine (source: WHO-2020, Expanded Program on Immunization, Pakistan; WHO-2019, Fact sheet Bangladesh). b n is the number of doses due for each antigen for the 6241 immunization visits recorded for 4613 children from both Pakistan and Bangladesh sites.

Participant Characteristics
A total of 16 vaccinators participated in the key informant interviews (Table 3). In Pakistan, 1 (9%) vaccinator out of 11 was female, whereas in Bangladesh, the proportion of female vaccinators was 80% (4/5). On average, the participants had been working as vaccinators for 7.6 (SD 8.4) years, and the mean age was 32 (SD 9.3) years. In Pakistan, nearly half (6/11, 54%) of all the vaccinators interviewed had >12 years of education, whereas in Bangladesh, this proportion was 100% (5/5).
We extracted 3 major themes through thematic analysis of in-depth interviews conducted with the vaccinators.

Theme 1: Understanding of iDSS
All (16/16, 100%) vaccinators appreciated the iDSS module for calculating the age-appropriate vaccine schedules for each visit. They relayed good knowledge of iDSS functionality and 69% (11/16) believed that they could effortlessly explain the app features to their fellow colleagues. Furthermore, most (13/16, 81%) vaccinators were aware of the color scheme and were able to interpret it correctly:

The color codes give us indications; vaccines that are supposed to be administered on current visit have different color codes, and already vaccinated have a different color code. [Vaccinator 3]
On assessing the knowledge about key variables on which the iDSS algorithm functions, very few vaccinators (6/16, 37%) were aware that the iDSS predicted future dates using the variables of date of birth, age, and immunization history.

Automatic Construction of Age-Appropriate Vaccine Schedules
All the vaccinators (16/16, 100%) appreciated the instantaneous calculation of age-appropriate schedules using the iDSS. They noted and were able to explain the significance of dates that appeared against each vaccine that was due on the current visit, given previously, or needed to be scheduled for an upcoming visit:

Accuracy in Generating Age-Appropriate Vaccine Schedules
The vaccinators conveyed mixed responses regarding the perceived accuracy of the iDSS algorithm across both the sites. Less than half (7/16, 44%) of the vaccinators stated that they found no discrepancies in the schedules constructed by the iDSS, whereas others stated discrepancies that contradicted their routine practices.
Despite endorsing the accuracy, 62% (10/16) participants across both sites were confused about the age-appropriate administration of IPV and measles vaccines. For instance, 54% (6/11) vaccinators in Pakistan were unsure about administering the IPV vaccine due at the age of 14 weeks according to the national immunization schedule. Some vaccinators believed that IPV should be administered with the third dose of Penta, irrespective of the age of the child, and hence considered iDSS to have scheduled the IPV dose inaccurately:

It happens that if a child's age is let's say 3 months or 3.5 months and he comes in for BCG, application shows IPV to be given as well at 3.5 months, but we don't do so and schedule IPV for future. [Vaccinator 1]
One of the participants from Pakistan (1/11, 9%) also questioned the interval between the 2 doses of measles, particularly for children who were behind their routine schedules:

We have some confusions, for example if a child's age is 12 months, then it (iDSS) gives date randomly and does not keep a gap of 3 months between two doses. [Vaccinator 8]
Vaccinators in Bangladesh (3/5, 60%) were unsure about the scheduling of upcoming vaccines that are not dependent on previous doses (for instance, the first dose of measles vaccine is not dependent on pentavalent-3 dose and should be administered when a child turns 9 months). Vaccinators believed that the measles vaccine should be "locked" on the iDSS (ie, not allowed to be given) until the child receives the pentavalent-3 dose. The same logic was applied for the IPV-2 dose due at the age of 14 weeks in Bangladesh:

Utility of the iDSS for Vaccinators
More than half (9/16, 56%) of the vaccinators stated that the iDSS is easy to use and 69% (11/16)  Almost one-fifth (3/16, 19%) of the vaccinators thought that iDSS would be more useful for outreach activities in which they often encounter children who have missed their routine vaccination doses: Though our country has a wide range of EPI centers, during our outreach activities we try to reach defaulter children and ensure to give them all required vaccines displayed as due vaccines on this app [iDSS]. [Vaccinator 13] Vaccinators (6/16, 37%) also suggested that iDSS should serve as a channel to digitalize immunization information systems and replace the conventionally used manual calculation methods for constructing vaccine schedules: Our country is going to digitalize the vaccination system and we are still dependent on the old hardcopy methods. It was needed in the past since we have started EPI activity. It will assist in data centralization. [Vaccinator 13]

Overall Feedback on iDSS
Overall, the vaccinators were satisfied with the app functionality and provided positive feedback about the app: In my experience iDSS is far better than our current system. Moreover, auto generation of vaccine schedules helps us to enter data within no time.
[ Vaccinator 14] Vaccinators (7/16, 44%) suggested that the color scheme should be reconsidered: There are no issues with color scheme and its well understood; however, if red could be replaced by any other color it makes us feel better. You know red is used to indicate danger. [Vaccinator 4] All vaccinators admired the current iDSS interface in terms of color scheme, text font and size, and design. Vaccinators mentioned that there was room for improvement, but they did not provide specific details of the features that needed to be changed:

Theme 4: Challenges With the Manual Calculation of Catch-up Schedules
As part of their routine work, vaccinators were mainly dependent on vaccination cards or parental recall to infer the date of birth of children and their vaccination history to manually calculate immunization schedules: If someone has lost the EPI card then we try to figure out through a recall method and ask the parent to tell us when the child gets the first vaccine. Then we ask for the date of birth, which they usually don't know, and tell us the age of the child. In this way, based on verbal discussion and age of child, for example, they say 2 or 3.5 months, we take the risk and vaccinate the child for 2nd or 3rd dose accordingly. [Vaccinator 11] For follow-up visits, vaccinators mentioned that the usual practice was to provide a calendar date with a 1-month gap for the next dose instead of following the standard 28-day interval between scheduling future doses as per the WHO-recommended EPI schedule guidelines. Vaccinators mentioned that to schedule doses with a 28-day interval, they had to refer to calendar and count the days to find the exact dates. A few times, they did not account for weekends when scheduling the subsequent visits. Vaccinators found iDSS useful in providing schedules automatically for subsequent visits that were in line with the WHO EPI guidelines: Previously, we had to see the dates one by one and calculate it, now the benefit is we just enter data and get dates automatically. It also shows the schedule for my next visit. [Vaccinator 5]

Principal Findings
Our study has demonstrated the diagnostic accuracy and feasibility of iDSS to accurately schedule age-appropriate vaccination doses across a multicountry LMIC setting. We also demonstrated the potential of iDSS to reduce MOVs caused by poor vaccinator adherence to standard immunization schedules and the interpretation of complex immunization schedules. Our study reports positive feedback received from a diverse set of vaccinators with varying levels of education and experience on the usability and acceptability of the iDSS.
Overall, the iDSS had higher accuracy in constructing age-appropriate immunization schedules as per the WHO-recommended EPI guidelines compared with vaccinators. Our study highlights the issue of missed vaccine doses despite children making contact with vaccinators. Reducing MOVs caused by complexity and changes in EPI schedules can improve immunization coverage, timeliness, and equity [14]. One study from the United States estimated that the potentially achievable vaccination coverage for 4+Diphtheria, tetanus, and acellular pertussis vaccine, 4+PCV, and full series of Haemophilus influenzae type b for children aged 19 to 35 months would have been 90% if missed opportunities in the administration of these vaccines were eliminated [26]. From an equity perspective, iDSS can enable children from disadvantaged backgrounds, who are already delayed on their vaccination, to catch-up with the rest through accurate administration of all due vaccine doses when they encounter a vaccinator.
Our findings show that the highest MOVs across Pakistan and Bangladesh sites were for polio and measles vaccines, as 27.5% (383/1391) and 5.3% (28/527) of all due immunization doses, respectively, were missed by vaccinators. A systematic review of MOVs across LMICs also reported the prevalence rate of MOVs with regard to polio vaccines to range from 13.4% to 46.7% [2]. According to the WHO-recommended EPI schedule, Pakistan and Bangladesh administer the IPV vaccine to all children at 14 weeks. However, feedback from vaccinators across both sites revealed that vaccinators considered it to be given alongside vaccines administered with Penta-3 (irrespective of age). This practice delayed the IPV vaccination, as the Penta, OPV, and PCV vaccines depend on previous doses and are often administered beyond 14 weeks of age [27]. These MOVs, especially for polio vaccines, are alarming, as they impede global efforts to eradicate polio. Pakistan remains one of the last 2 polio-endemic countries with a substantial surge in the number of polio cases since 2017 (147 cases in 2019 vs 8 in 2017) [28], putting the country on a failing trajectory and the broader Global Polio Eradication Initiative at risk. Similarly, the measles incidence in the country increased from 24.6 cases per million to 80.4 per million between 2000 and 2018 [29].
Although previous studies have reported examples of EIR in LMIC settings [30,31], not all of them have an in-built immunization scheduling or decision support system [32][33][34].
Our study adds to this important area by providing evidence of the diagnostic accuracy of a stand-alone mobile-based iDSS implemented across a multicountry LMIC setting. Our results are in line with findings reported from high-income countries where iDSS has improved the scheduling of vaccinations. A hospital-based study in the United States that implemented a computer-based clinical decision support algorithm demonstrated an increase in the tetanus, diphtheria, pertussis vaccination in postpartum women [35]. Our results corroborate this finding, making a strong case for high diagnostic accuracy of iDSS technology in LMIC settings. In addition, the end-user feedback from a diverse set of vaccinators, who varied in terms of gender, education, age, and experience, also confirms the utility and functionality of iDSS, its acceptance, and user satisfaction; however, interviews with vaccinators in our study did reveal that only about half (9/16, 56%) of them found the iDSS easy to use, presumably because of maintaining paper-based and electronic records simultaneously in the study. Switching to electronic records completely is likely to address this concern [14].
In contrast to most web-based iDSS developed and used in high-income countries, the iDSS in this study is packaged in the form of an API that is interoperable with health information systems and allows flexibility of deployment. As EIRs continue to be adapted, incorporating iDSS through APIs should be considered, in line with established standards such as the Pan American Health Organization EIR guidelines [36]. API-based iDSS such as the one used in this study also allows flexibility to adjust to multicountry EPI schedules that change frequently and can also adapt to cosmetic user interface changes such as varying languages and displays.
Integration of an iDSS module in LMIC Immunization Programs can, therefore, yield enormous benefits, not only in terms of reducing MOVs but also in designing accurate catch-up regimens to ensure universal immunization. This is particularly relevant in the context of the huge numbers of children who have missed their vaccine doses during the COVID-19 pandemic and associated lockdowns [18]. Equipping vaccinators with iDSS technology can help ensure that these children are effectively immunized for all due vaccines at subsequent visits, thereby maintaining optimal immunity levels and reducing the likelihood of secondary vaccine-preventable disease outbreaks.

Strengths and Limitations
A major strength of our study was the demonstration of the feasibility of the iDSS across 2 different LMIC settings, with varying EPI schedules, infrastructure, coverage levels, vaccinator education, experience, demographics, and immunization-related challenges. One limitation of our study was that the evidence of MOVs generated in this study only accounted for MOVs because of inaccurate vaccination schedules constructed by vaccinators and hence would be an underestimation of the overall MOV prevalence in our study sites.

Conclusions
The iDSS has high diagnostic accuracy for scheduling age-appropriate vaccinations and reducing MOVs; and high acceptability among vaccinators regardless of gender, education, and experience. The iDSS boasts a variety of features for easy adaptability and replicability across LMIC settings. The evidence generated from this study demonstrates the prevalence of MOVs, especially for measles and polio vaccines. iDSS can increase immunization coverage, timeliness, and equity by eliminating these MOVs and help design accurate catch-up regimens to ensure universal immunization, especially in the aftermath of the COVID-19 pandemic. The findings from this study provide the impetus for rigorously evaluating the impact of iDSS through a randomized controlled trial and paving the way for a scaled implementation of this tool across LMICs.